\chapter{Running EnKF}\label{enkf_run}
\setlength{\parskip}{12pt}

%----------------------------------------------
\section{Input Data Required to Run EnKF}
%----------------------------------------------

This chapter discusses the process of running EnKF cases. It includes:
\begin{enumerate}
\item Discussions of the input data required to run EnKF.
\item A detailed explanation of how to run both a regional and global EnKF with the released run scripts. 
\item Introduction to the files in a successful regional and global EnKF run directory
\end{enumerate}

%----------------------------------------------
\subsection{Input Data Required to Run EnKF}
%----------------------------------------------

In most cases, three types of input data, ensemble mean and members, observations, 
and fixed files, must be available before running EnKF.\\

\textbf{1. Ensemble mean and members} \\
The ensemble members and ensemble mean of certain regional and global ensemble 
forecast systems are used as the background for the EnKF analysis. When this EnKF 
system is used for regional analysis with WRF ensembles, the ensemble members 
and ensemble mean follow the naming convention:
\begin{verbatim}
firstguess.mem001 
firstguess.mem002
... ...
firstguess.ensmean
\end{verbatim}

Please note that the number of allocated computer cores to run EnKF must be larger than the ensemble size. 
The ensemble members can be generated using various methods, such as:
\begin{itemize}
\item Using global/regional ensemble forecasts.
\item Ensemble forecasts generated using multi-physics, multi-models, or adding
random perturbations drawn from climatology.
\item In cycling assimilation, using ensemble forecasts initialized from previous ensemble analyses generated by EnKF. 
\end{itemize}

This version EnKF can use any of the following ensemble files as the background:
\begin{itemize}
\item ARW NetCDF forecast
\item NMM NetCDF forecast
\item GFS forecast files
\end{itemize}

\textbf{2. Prepare observation ensemble priors (observation innovation)} \\
In addition to the ensemble backgrounds on model grids, the ensemble priors of all observations 
(observation innovation for all ensemble members) are also needed to run EnKF. The observation 
ensemble priors are generated by running GSI observation forward operators with the ensemble 
members as backgrounds (without doing actual GSI analyses). In this release, the GSI v3.5 run 
script includes options for generating the observation ensemble priors (details in section 3.2). 
The observation ensemble priors files should follow the following naming conventions:
\begin{itemize}
\item For conventional observations:
\begin{verbatim}
diag_conv_ges.mem001 
diag_conv_ges.mem002
... ...
diag_conv_ges.ensmean
\end{verbatim}
\item For radiance observations:
\begin{verbatim}
diag_instrument_Satellite.mem001 (e.g. diag_hirs4_n19.mem001) 
diag_ instrument_Satellite.mem002 (e.g. diag_hirs4_n19.mem002)
... ...
diag_ instrument_Satellite.ensmean (e.g. diag_hirs4_n19.ensmean)
\end{verbatim}
\end{itemize}

These diag files contain a lot of information about each observation. For more details on the 
content of diag files, please refer to the GSI User\textquotesingle s Guide Appendix A.2.
The preparation of observations for EnKF assimilation is done within GSI, including quality control 
of observations, selection of observation types for assimilation, and observation error tuning. In 
the default namelist situation, NO additional online quality control of observations is performed in 
the EnKF analysis step (although there is an option to do the similar quality control of observations 
as the GSI variational scheme.)

\textbf{3. Fixed files} \\
EnKF uses the the same fixed files as GSI to setup the analysis configurations. Detailed explanation 
of all fixed files provided in the community GSI system is in the GSI user\textquotesingle s Guide, Chapter 3. 
The following is a list of fixed files needed for EnKF analyses:
\begin{itemize}
\item for observation control:
\begin{verbatim}
convinfo - conventional data (prepufr) info file 
ozinfo - ozone retrieval info file
satinfo - satellite channel info file
\end{verbatim}
\item when satellite radiance data are assimilated, the following files are needed to do the adaptive radiance bias correction:
\begin{small}
\begin{verbatim}
satbias_in - satellite bias correction coefficient file
satbias_pc - satellite bias correction coefficient file for passive channels
\end{verbatim}
\end{small}
\end{itemize}

Note that this version EnKF uses adaptive bias correction. The bias correction coefficients are in a single file that combined satellite angle dependent and mass bias correction coefficients. See GSI User\textquotesingle s Guide for 
more detail. When the namelist parameter \textit{readin\_localization} is set to true, file \textit{hybens\_locinfo} is 
needed, in which customized localization values varying by model level are contained.

%----------------------------------------------
\section{EnKF and GSI Observer Run Scripts}
%----------------------------------------------

In this release version, four sample run scripts for EnKF applications are under directory 
\verb|comGSIv3.5_EnKFv1.1/run|:
\begin{itemize}
\item \textit{run\_gsi\_regional.ksh} for running regional GSI to generate the observation ensemble priors. Referred to as the GSI observer run scripts.
\item \textit{run\_gsi\_global.ksh} is GSI observer run script for global applications (to generate the observation ensemble priors).
\item \textit{run\_enkf\_wrf.ksh} for running regional EnKF
\item \textit{run\_enkf\_global.ksh} for running global EnKF
\end{itemize}

These run scripts are introduced in detail in the following sections. Also provided are two scripts for generating the GSI and EnKF namelist:
\begin{itemize}
\item \textit{comgsi\_namelist.sh} generates GSI namelist on the fly (called by GSI observer run scripts).
\item \textit{comgsi\_namelist\_gfs.sh} generates GSI namelist on the fly (called by GSI observer run scripts) for global GSI applications. 
\item \textit{enkf\_wrf\_namelist.sh} generates EnKF namelist on the fly (called by EnKF run script) for regional EnKF applications.
\end{itemize}

%----------------------------------------------
 \subsection{General Introduction to the Run Scripts}
%----------------------------------------------

These run scripts provide the run time environment necessary for running the GSI and EnKF executables. They all have similar steps, as follows:
\begin{enumerate}
\item Request computer resources to run GSI/EnKF.
\item Set environmental variables for the machine architecture.
\item Set experimental variables (such as experiment name, analysis time, background, and observation).
\item Check the definitions of required variables. 
Generate a run directory for GSI/EnKF
\item Copy the GSI/EnKF executable to the run directory.
\item Copy/Link the background file/ensemble files to the run directory.
\item Link observations to the run directory.
\item Link fixed files (statistic, control, and coefficient files) to the run directory.
\item  Generate namelist for GSI/EnKF.
\item Run the GSI/EnKF executable.
\item Post-process: save analysis results, generate diagnostic files, clean run directory.
\end{enumerate}

In the GSI User\textquotesingle s Guide, three sections explain the first three steps in detail:
\begin{itemize}
\item Section 3.2.2.1: Setting up the machine environment (step 1)
\item Section 3.2.2.2: Setting up the running environment (step 2)
\item Section 3.2.2.3: Setting up an analysis case (step 3)
\end{itemize}

For this documentation, the first 2 steps will be skipped and the 3rd step in the GSI observer 
and EnKF run scripts will be discussed. The community GSI analysis run script and the GSI 
observer run script are the same, with the GSI observer capability controlled by flags 
that turn off the minimization, select appropriate namelist options and enable looping through 
all the ensemble members to generate the ensemble observation priors for each member, 
including the ensemble mean.
   
%----------------------------------------------
\subsection{GSI Observer Run Scripts}
%----------------------------------------------

      \subsubsection{Setting up a case}
      This section discusses variables specific to the user\textquotesingle s case, such as analysis time, working directory, background and observation files, location of fixed files and CRTM coefficients, and the GSI executable file. The script looks like:
\begin{scriptsize}
 \begin{verbatim}
##################################################### 
# case set up (users should change this part) 
##################################################### 
#
# ANAL_TIME= analysis time  (YYYYMMDDHH)
# WORK_ROOT= working directory, where GSI runs
# PREPBURF = path of PreBUFR conventional obs
# BK_FILE  = path and name of background file
# OBS_ROOT = path of observations files
# FIX_ROOT = path of fix files
# GSI_EXE  = path and name of the gsi executable
  ANAL_TIME=2014061700
  HH=`echo $ANAL_TIME | cut -c9-10`
  WORK_ROOT=run/testarw
  OBS_ROOT=data/20140617/obs
  PREPBUFR=${OBS_ROOT}/nam.t${HH}z.prepbufr.tm00.nr
  BK_ROOT=data/20140617/2014061700/arw
  BK_FILE=${BK_ROOT}/wrfinput_d01.${ANAL_TIME}
  CRTM_ROOT=data/CRTM_2.2.3
  GSI_ROOT=code/comGSIv3.5_EnKFv1.1
  FIX_ROOT=${GSI_ROOT}/fix
  GSI_EXE=${GSI_ROOT}/run/gsi.exe
  GSI_NAMELIST=${GSI_ROOT}/run/comgsi_namelist.sh
\end{verbatim}
\end{scriptsize}

The options ANAL\_TIME, WORK\_ROOT, PREPBURF, BK\_FILE, OBS\_ROOT, FIX\_ROOT, GSI\_EXE are all the same settings as the GSI analysis configuration. Two options: BK\_ROOT , GSI\_ROOT, are the root directories for ensemble members and the GSI system. These exist to make links to the background and GSI system easy and shorter. The new option: GSI\_NAMELIST is needed because the namelist section was taken out of the run scripts in this release as a separate file to improve the structure and readability of the run scripts. Users can find the namelist files for both GSI and EnKF in the same directory as the run scripts. Please note the option BK\_FILE is pointing to the ensemble mean.
The next part of this block has additional options to specify other important aspects of the GSI observer.
\begin{scriptsize}
 \begin{verbatim}
 #---------------------
# bk_core= which WRF core is used as background (NMM or ARW or NMMB)
# bkcv_option= which background error covariance and parameter will be used
#              (GLOBAL or NAM)
# if_clean = clean  : delete temperal files in working directory (default)
#            no     : leave running directory as is (this is for debug only)
  bk_core=ARW
  bkcv_option=NAM
  if_clean=clean
# if_observer = Yes  : only used as observation operater for enkf
# no_member     number of ensemble members
# BK_FILE_mem   path and base for ensemble members
  if_observer=No   # Yes, or, No -- case sensitive !
  no_member=20
  BK_FILE_mem=${BK_ROOT}/wrfarw.mem
\end{verbatim}
\end{scriptsize}
  
  The options \textit{bk\_core}, \textit{bkcv\_option}, and, \textit{if\_clean} are the same as in the GSI analysis run scripts. The new option if\_observer indicates if this GSI run is for the generation of the observation ensemble priors or for a regular GSI run. The new option \textit{no\_member} specifies the number of ensemble members that are need to be calculated for the observation ensemble priors. This should also be the ensemble number in the EnKF analysis. Option \textit{BK\_FILE\_mem} is the path and the name of the ensembles without the ensemble member ID appended. The scripts will add the ensemble member ID as a three digital number, such as 000, 001,\ldots .
      
\subsubsection{Loop through ensemble members}
 As mentioned previously, the GSI ensemble observer run scripts are the same as the GSI analysis run scripts released with the community GSI. Since the observer only generates diag files, which includes useful information on the observation innovation, the GSI outer loop number for the observer should be set to 0 to skip all minimization iterations.
 
 The contents of the run scripts can be divided into two parts, those before the following comments and those after:
 \begin{scriptsize}
 \begin{verbatim}
#################################################
# start to calculate diag files for each member
#################################################
 \end{verbatim}
 \end{scriptsize}
 
Before this comment, the scripts have the same functionality as when running a GSI analysis, except that the background options in the scripts for the observer funcationality are set for the ensemble mean. Additionally, the namelist is built with the two following options set, which skips the minimization and saves all observation processing from the ensemble mean:
\begin{scriptsize}
\begin{verbatim}
if [ ${if_observer} = Yes ] ; then
  nummiter=0
  if_read_obs_save='.true.'
  if_read_obs_skip='.false.'
else
\end{verbatim}
\end{scriptsize}

Please refer to the GSI user\textquotesingle s guide for detailed explanation of the remainder of this portion of the run scripts. The second portion of the script loops through each member to calculate the observation ensemble priors based on the GSI run environments setup by the first portion.

Listed below is an annotated version of the 2nd part of the GSI observer run script with explanations on each function block.
\begin{scriptsize}
\begin{verbatim}
if [ ${if_observer} = Yes ] ; then
\end{verbatim}
\end{scriptsize}

This 2nd part of the script only runs if option \textit{if\_observer} is set to "Yes". The diag files from the ensemble mean need to be saved first with the following commands:
\begin{scriptsize}
\begin{verbatim}
  string=ges
  for type in $listall; do
    count=0
    if [[ -f diag_${type}_${string}.${ANAL_TIME} ]]; then
       mv diag_${type}_${string}.${ANAL_TIME} diag_${type}_${string}.ensmean
    fi
  done
  mv wrf_inout wrf_inout_ensmean
\end{verbatim}
\end{scriptsize}

The following section builds the namelist for ensemble members. Please note two options need to be set different between mean and members:
\begin{scriptsize}
\begin{verbatim}
# Build the GSI namelist on-the-fly for each member
  nummiter=0
  if_read_obs_save='.false.'
  if_read_obs_skip='.true.'
. $GSI_NAMELIST
cat << EOF > gsiparm.anl

 $comgsi_namelist

EOF
\end{verbatim}
\end{scriptsize}

The option \textit{if\_read\_obs\_save} and \textit{if\_read\_obs\_skip} switch from "True" and "False", respectively for the mean to "False" and "True", respectively for the ensemble members.

This saves all observation processing information (including bias correction, thinning, etc) from the ensemble mean and save the same information for the ensemble members to keep observations  constant.

The script loops through each ensemble member (from member 001 to \textit{no\_member}) to create the diag files for each member:

\begin{scriptsize}
\begin{verbatim}
# Loop through each member
  loop="01"
  ensmem=1
  while [[ $ensmem -le $no_member ]];do

     rm pe0*

     print "\$ensmem is $ensmem"
     ensmemid=`printf %3.3i $ensmem`
\end{verbatim}
\end{scriptsize}

After a member is processed, the script removes the old ensemble member and links to the new member before rerunning the calculation:
\begin{scriptsize}
\begin{verbatim}
# get new background for each member
     if [[ -f wrf_inout ]]; then
       rm wrf_inout
     fi

     BK_FILE=${BK_FILE_mem}${ensmemid}
     echo $BK_FILE
     ln -s $BK_FILE wrf_inout
\end{verbatim}
\end{scriptsize}

Run the GSI observer for this member:
\begin{scriptsize}
\begin{verbatim}
#  run  GSI
     echo ' Run GSI with' ${bk_core} 'for member ', ${ensmemid}

     case $ARCH in
        'IBM_LSF')
           ${RUN_COMMAND} ./gsi.exe < gsiparm.anl > stdout_mem${ensmemid} 2>&1  ;;

        * )
           ${RUN_COMMAND} ./gsi.exe > stdout_mem${ensmemid} 2>&1 ;;
     esac

#  run time error check and save run time file status
     error=$?

     if [ ${error} -ne 0 ]; then
       echo "ERROR: ${GSI} crashed for member ${ensmemid} Exit status=${error}"
       exit ${error}
     fi

     ls -l * > list_run_directory_mem${ensmemid}
\end{verbatim}
\end{scriptsize}

Generate diag files for this member:
\begin{scriptsize}
\begin{verbatim}
# generate diag files

     for type in $listall; do
           count=`ls pe*${type}_${loop}* | wc -l`
        if [[ $count -gt 0 ]]; then
           cat pe*${type}_${loop}* > diag_${type}_${string}.mem${ensmemid}
        fi
     done

# next member
     (( ensmem += 1 ))

  done

fi
\end{verbatim}
\end{scriptsize}

Since all members are using the same run directory, the run status of each member is overwriten by the following member. The stdout file and the file list in run directory are preserved with the ensemble member ID for debug.

%----------------------------------------------
\subsection{Sample Regional EnKF Run Scripts}
%----------------------------------------------

As described in section 3.2.1, the regional EnKF run scripts have been designed to have a similar structure to the GSI analysis and observer run scripts. Again, please refer to the GSI User\textquotesingle s Guide section 3.2.2.1 and 3.2.2.2 for the first two steps.

The 3rd step is to setup the variables specific to the user\textquotesingle s case, such as analysis time, working directory, background and observation files, location of fixed files and CRTM coefficients, and the EnKF executable. Most of the options in this portion are the same as the 3rd step in the GSI observer run scripts, which is discussed in section 3.2.2 of this User\textquotesingle s Guide. Users should setup most of the variables in this portion based on the options in the GSI observer run scripts as they are the variables to setup the same things for the GSI and EnKF. The following is a sample script with explanations:
\begin{scriptsize}
\begin{verbatim}
#
#####################################################
# case set up (users should change this part)
#####################################################
#
# ANAL_TIME= analysis time  (YYYYMMDDHH)
# WORK_ROOT= working directory, where GSI runs
# PREPBURF = path of PreBUFR conventional obs
# BK_FILE  = path and name of background file
# OBS_ROOT = path of observations files
# FIX_ROOT = path of fix files
# GSI_EXE  = path and name of the gsi executable

  ANAL_TIME=2012102506
  WORK_ROOT=enkf/regional/enkf_arw
  diag_ROOT=enkf/regional/gsidiag_arw
  BK_ROOT=enkf/enkfdata/arw/bk
  BK_FILE=${BK_ROOT}/wrfarw.ensmean
  GSI_ROOT=/enkf/code/comGSIv3.4_EnKFv1.0
  FIX_ROOT=${GSI_ROOT}/fix
  ENKF_EXE=${GSI_ROOT}/src/main/enkf/wrf_enkf
  CRTM_ROOT=CRTM_REL-2.2.3
  ENKF_NAMELIST=${GSI_ROOT}/run/enkf_wrf_namelist.sh
\end{verbatim}
\end{scriptsize}

Options \textit{ANAL\_TIME}, \textit{BK\_ROOT}, \textit{BK\_FILE}, \textit{GSI\_ROOT}, \textit{FIX\_ROOT}, \textit{CRTM\_ROOT} have 
the same meanings as the ensemble GSI observer run scripts and should be set to the 
same values as the ensemble GSI observer run scripts. The option \textit{WORK\_ROOT} is 
the working directory which should have enough space to hold the ensemble members 
and EnKF analysis results. The option \textit{diag\_ROOT} is pointing to the run directory of the 
GSI observer, where the diag files are generated as data input for the EnKF. The option 
\textit{ENKF\_EXE} points to the EnKF executable, which is under the GSI source code directory 
in this release. The option \textit{ENKF\_NAMELIST} is the path and the EnKF namelist file, which 
sits outside of the run script as a separate file like the GSI namelist. Users can find the 
namelist files for both GSI and the EnKF in the same directory as the run scripts.

The next part of this block includes several additional options that specify several aspects of the ensemble members.

\begin{scriptsize}
\begin{verbatim}
# ensemble parameters
#
  NMEM_ENKF=20
  BK_FILE_mem=${BK_ROOT}/wrfarw
  NLONS=111
  NLATS=111
  NLEVS=56
  IF_ARW=.true.
  IF_NMM=.false.
  list="conv amsua_n18 hirs4_n19"
\end{verbatim}
\end{scriptsize}

Options \textit{NMEM\_ENKF}, \textit{BK\_FILE\_mem} are also in the GSI observer run script and 
should be set to the same values as the GSI observer run script. Options \textit{NLONS}, \textit{NLATS}, 
and \textit{NLEVS} specify 3 dimensions (XYZ ) of the ensemble grid. Options \textit{IF\_ARW} and 
\textit{IF\_NMM} indicates which background, ARW NetCDF ensemble or the NMM NetCDF 
ensemble, is used in this EnKF run. Option \textit{list} is a list of the observation types that the 
EnKF will use in the analysis. This list should be based on the diag files generated by 
the ensemble GSI observer.

At this point, users should be able to run the EnKF for simple cases without changing the rest of the script. However, some advanced users may need to change some of the following blocks for special applications.
\begin{scriptsize}
\begin{verbatim}
#####################################################
# Users should NOT change script after this point
#####################################################
\end{verbatim}
\end{scriptsize}

The next block sets the run command to run EnKF on multiple platforms. The \textit{ARCH} is set at the beginning of the script.

\begin{scriptsize}
\begin{verbatim}
case $ARCH in
   'IBM_LSF')
      ###### IBM LSF (Load Sharing Facility)
      RUN_COMMAND="mpirun.lsf " ;;

   'LINUX')
      if [ $GSIPROC = 1 ]; then
         #### Linux workstation - single processor
         RUN_COMMAND=""
      else
         ###### Linux workstation -  mpi run
        RUN_COMMAND="mpirun -np ${GSIPROC} -machinefile ~/mach "
      fi ;;

   'LINUX_LSF')
      ###### LINUX LSF (Load Sharing Facility)
      RUN_COMMAND="mpirun.lsf " ;;

   'LINUX_PBS')
      #### Linux cluster PBS (Portable Batch System)
#      RUN_COMMAND="mpirun -np ${GSIPROC} " ;;
      RUN_COMMAND="mpiexec_mpt -n ${GSIPROC} " ;;

   'DARWIN_PGI')
      ### Mac - mpi run
      if [ $GSIPROC = 1 ]; then
         #### Mac workstation - single processor
         RUN_COMMAND=""
      else
         ###### Mac workstation -  mpi run
         RUN_COMMAND="mpirun -np ${GSIPROC} -machinefile ~/mach "
      fi ;;

   * )
     print "error: $ARCH is not a supported platform configuration."
     exit 1 ;;
esac
\end{verbatim}
\end{scriptsize}


The following block sets up fixed files and some analysis-time related values:
\begin{scriptsize}
\begin{verbatim}
# Given the analysis date, compute the date from which the
# first guess comes.  Extract cycle and set prefix and suffix
# for guess and observation data files
# gdate=`$ndate -06 $adate`
gdate=$ANAL_TIME
YYYYMMDD=`echo $adate | cut -c1-8`
HH=`echo $adate | cut -c9-10`

# Fixed files
# CONVINFO=${FIX_ROOT}/global_convinfo.txt
# SATINFO=${FIX_ROOT}/global_satinfo.txt
# SCANINFO=${FIX_ROOT}/global_scaninfo.txt
# OZINFO=${FIX_ROOT}/global_ozinfo.txt
CONVINFO=${diag_ROOT}/convinfo
SATINFO=${diag_ROOT}/satinfo
SCANINFO=${diag_ROOT}/scaninfo
OZINFO=${diag_ROOT}/ozinfo
# LOCINFO=${FIX_ROOT}/global_hybens_locinfo.l64.txt
\end{verbatim}
\end{scriptsize}

The next block creates a working directory (\verb|workdir|) in which EnKF will run. The directory should have enough disk space to hold all the files needed for this run. This directory is cleaned before each run, therefore, save all the files needed from the previous run before rerunning EnKF.
\begin{scriptsize}
\begin{verbatim}
# Set up workdir
rm -rf $WORK_ROOT
mkdir -p $WORK_ROOT
cd $WORK_ROOT
\end{verbatim}
\end{scriptsize}

After creating a working directory, copy or link the EnKF executable, ensembles, diag files (observations), bias correction coefficients, and fixed files into the working directory.
\begin{scriptsize}
\begin{verbatim}
cp $ENKF_EXE        ./enkf.x

cp $CONVINFO        ./convinfo
cp $SATINFO         ./satinfo
cp $SCANINFO        ./scaninfo
cp $OZINFO          ./ozinfo
# cp $LOCINFO         ./hybens_locinfo

cp $diag_ROOT/satbias_in ./satbias_in
cp $diag_ROOT/satbias_pc ./satbias_pc

# get mean
ln -s ${BK_FILE_mem}.ensmean ./firstguess.ensmean
for type in $list; do
   ln -s $diag_ROOT/diag_${type}_ges.ensmean .
done

# get each member
imem=1
while [[ $imem -le $NMEM_ENKF ]]; do
   member="mem"`printf %03i $imem`
   ln -s ${BK_FILE_mem}.${member} ./firstguess.${member}
   for type in $list; do
      ln -s $diag_ROOT/diag_${type}_ges.${member} .
   done
   (( imem = $imem + 1 ))
done
\end{verbatim}
\end{scriptsize}

The following script is used to generate the EnKF namelist called enkf.nml in the working directory. Some namelist variables are explained in detail in Section 4.3. Appendix A gives a full list of namelist options.
\begin{scriptsize}
\begin{verbatim}
# Build the GSI namelist on-the-fly
. $ENKF_NAMELIST
cat << EOF > enkf.nml

 $enkf_namelist

EOF
\end{verbatim}
\end{scriptsize}

Copy the ensemble background files to the working directory and rename them as \verb| "analysis.${member}" |. The EnKF will update those files as the analysis results.

\begin{scriptsize}
\begin{verbatim}
# make analysis files
cp firstguess.ensmean analysis.ensmean
# get each member
imem=1
while [[ $imem -le $NMEM_ENKF ]]; do
   member="mem"`printf %03i $imem`
   cp firstguess.${member} analysis.${member}
   (( imem = $imem + 1 ))
done
\end{verbatim}
\end{scriptsize}

The following block runs EnKF and checks if the EnKF has successfully completed.
\begin{scriptsize}
\begin{verbatim}
###################################################
#  run  EnKF
###################################################
echo ' Run EnKF'

${RUN_COMMAND} ./enkf.x < enkf.nml > stdout 2>&1

##################################################################
#  run time error check
##################################################################
error=$?

if [ ${error} -ne 0 ]; then
  echo "ERROR: ${ENKF_EXE} crashed  Exit status=${error}"
  exit ${error}
fi
\end{verbatim}
\end{scriptsize}

If this point is reached, the EnKF successfully finishes and exits with 0:
\begin{scriptsize}
\begin{verbatim}
exit
\end{verbatim}
\end{scriptsize}


%----------------------------------------------
\section{Understanding Resulting Files in GSI Observer and EnKF Run Directory}
%----------------------------------------------

To check if the GSI observer and EnKF runs have been successfully finished, it is important to understand the meaning of each file in the run directory.

%----------------------------------------------
\subsection{The GSI Observer Run Directory}
%----------------------------------------------

After customizing the GSI observer run script to your personal environment, it may be submitted to the batch system just as any other job.
Following a successful run, the majority of the files in the GSI observer run directory will be the same as those in a sucessful GSI analysis run directory. The difference for the observer run is that the GSI observer generates more diag and stdout files related to each ensemble member. Below is an example of the files generated in the run directory from a GSI observer run:
\begin{scriptsize}
\begin{verbatim}
amsuabufr                       diag_amsua_n18_ges.mem010       fit_t1.2014021300          mhsbufr
amsubbufr                       diag_amsua_n18_ges.mem011       fit_w1.2014021300          obs_input.0001
anavinfo                        diag_amsua_n18_ges.mem012       fort.201                   obs_input.0002
berror_stats                    diag_amsua_n18_ges.mem013       fort.202                   obs_input.0003
convinfo                        diag_amsua_n18_ges.mem014       fort.203                   obs_input.0004
diag_amsua_metop-a_ges.ensmean  diag_amsua_n18_ges.mem015       fort.204                   obs_input.0006
diag_amsua_metop-a_ges.mem001   diag_amsua_n18_ges.mem016       fort.205                   obs_input.0010
diag_amsua_metop-a_ges.mem002   diag_amsua_n18_ges.mem017       fort.206                   obs_input.0019
diag_amsua_metop-a_ges.mem003   diag_amsua_n18_ges.mem018       fort.207                   obs_input.0021
diag_amsua_metop-a_ges.mem004   diag_amsua_n18_ges.mem019       fort.208                   obs_input.0022
diag_amsua_metop-a_ges.mem005   diag_amsua_n18_ges.mem020       fort.209                   obs_input.0026
diag_amsua_metop-a_ges.mem006   diag_conv_ges.ensmean           fort.210                   obs_input.0027
diag_amsua_metop-a_ges.mem007   diag_conv_ges.mem001            fort.211                   obs_input.0028
diag_amsua_metop-a_ges.mem008   diag_conv_ges.mem002            fort.212                   obs_input.0029
diag_amsua_metop-a_ges.mem009   diag_conv_ges.mem003            fort.213                   obs_input.0030
diag_amsua_metop-a_ges.mem010   diag_conv_ges.mem004            fort.214                   obs_input.common
diag_amsua_metop-a_ges.mem011   diag_conv_ges.mem005            fort.215                   ozinfo
diag_amsua_metop-a_ges.mem012   diag_conv_ges.mem006            fort.217                   pcpbias_out
diag_amsua_metop-a_ges.mem013   diag_conv_ges.mem007            fort.218                   pcpinfo
diag_amsua_metop-a_ges.mem014   diag_conv_ges.mem008            fort.219                   prepbufr
diag_amsua_metop-a_ges.mem015   diag_conv_ges.mem009            fort.220                   prepobs_prep.bufrtable
diag_amsua_metop-a_ges.mem016   diag_conv_ges.mem010            fort.221                   satbias_ang.out
diag_amsua_metop-a_ges.mem017   diag_conv_ges.mem011            fort.223                   satbias_in
diag_amsua_metop-a_ges.mem018   diag_conv_ges.mem012            fort.224                   satbias_out
diag_amsua_metop-a_ges.mem019   diag_conv_ges.mem013            fort.225                   satbias_out.int
diag_amsua_metop-a_ges.mem020   diag_conv_ges.mem014            fort.226                   satbias_pc
diag_amsua_n15_ges.ensmean      diag_conv_ges.mem015            fort.227                   satbias_pc.out
diag_amsua_n15_ges.mem001       diag_conv_ges.mem016            fort.228                   satinfo
diag_amsua_n15_ges.mem002       diag_conv_ges.mem017            fort.229                   sigf03
diag_amsua_n15_ges.mem003       diag_conv_ges.mem018            fort.230                   stdout
diag_amsua_n15_ges.mem004       diag_conv_ges.mem019            gpsrobufr                  stdout.anl.2014021300
diag_amsua_n15_ges.mem005       diag_conv_ges.mem020            gsi.exe                    stdout_mem001
diag_amsua_n15_ges.mem006       diag_hirs4_metop-a_ges.ensmean  gsiparm.anl                stdout_mem002
diag_amsua_n15_ges.mem007       diag_hirs4_metop-a_ges.mem001   hirs3bufr                  stdout_mem003
diag_amsua_n15_ges.mem008       diag_hirs4_metop-a_ges.mem002   hirs4bufr                  stdout_mem004
diag_amsua_n15_ges.mem009       diag_hirs4_metop-a_ges.mem003   l2rwbufr                   stdout_mem005
diag_amsua_n15_ges.mem010       diag_hirs4_metop-a_ges.mem004   list_run_directory         stdout_mem006
diag_amsua_n15_ges.mem011       diag_hirs4_metop-a_ges.mem005   list_run_directory_mem001  stdout_mem007
diag_amsua_n15_ges.mem012       diag_hirs4_metop-a_ges.mem006   list_run_directory_mem002  stdout_mem008
diag_amsua_n15_ges.mem013       diag_hirs4_metop-a_ges.mem007   list_run_directory_mem003  stdout_mem009
diag_amsua_n15_ges.mem014       diag_hirs4_metop-a_ges.mem008   list_run_directory_mem004  stdout_mem010
diag_amsua_n15_ges.mem015       diag_hirs4_metop-a_ges.mem009   list_run_directory_mem005  stdout_mem011
diag_amsua_n15_ges.mem016       diag_hirs4_metop-a_ges.mem010   list_run_directory_mem006  stdout_mem012
diag_amsua_n15_ges.mem017       diag_hirs4_metop-a_ges.mem011   list_run_directory_mem007  stdout_mem013
diag_amsua_n15_ges.mem018       diag_hirs4_metop-a_ges.mem012   list_run_directory_mem008  stdout_mem014
diag_amsua_n15_ges.mem019       diag_hirs4_metop-a_ges.mem013   list_run_directory_mem009  stdout_mem015
diag_amsua_n15_ges.mem020       diag_hirs4_metop-a_ges.mem014   list_run_directory_mem010  stdout_mem016
diag_amsua_n18_ges.ensmean      diag_hirs4_metop-a_ges.mem015   list_run_directory_mem011  stdout_mem017
diag_amsua_n18_ges.mem001       diag_hirs4_metop-a_ges.mem016   list_run_directory_mem012  stdout_mem018
diag_amsua_n18_ges.mem002       diag_hirs4_metop-a_ges.mem017   list_run_directory_mem013  stdout_mem019
diag_amsua_n18_ges.mem003       diag_hirs4_metop-a_ges.mem018   list_run_directory_mem014  stdout_mem020
diag_amsua_n18_ges.mem004       diag_hirs4_metop-a_ges.mem019   list_run_directory_mem015  wrfanl.2014021300
diag_amsua_n18_ges.mem005       diag_hirs4_metop-a_ges.mem020   list_run_directory_mem016  wrf_inout
diag_amsua_n18_ges.mem006       errtable                        list_run_directory_mem017  wrf_inout_ensmean
diag_amsua_n18_ges.mem007       fit_p1.2014021300               list_run_directory_mem018
diag_amsua_n18_ges.mem008       fit_q1.2014021300               list_run_directory_mem019
diag_amsua_n18_ges.mem009       fit_rad1.2014021300             list_run_directory_mem020
\end{verbatim}
\end{scriptsize}

This case was a regional analysis with WRF/ARW NetCDF backgrounds. In this case, 20 ensemble members are used to generate the diag files. A brief introduction of the additional files in the GSI observer runs is given below:


\begin{itemize}[itemindent=-15pt]
\item \textit{diag\_conv\_ges.mem???}  - Diag files of conventional observations for ensemble member ???.
\item \textit{diag\_conv\_ges.ensmean}  - Diag files of satellite radiance observation for ensemble mean.
\item \textit{stdout\_mem???}  - Standard output from GSI observer run for ensemble member ???.
\item \textit{list\_run\_directory\_mem???} - The list of the files in the run directory after the GSI observer is finished for ensemble member ???.
\end{itemize}



%----------------------------------------------
\subsection{Files in the EnKF Run Directory}
%----------------------------------------------

 After customizing the EnKF run script to your personal environment, it may be submitted to the batch system just as any other job. Upon successful assimilation, the ensemble analyses (both members and ensemble mean), covariance inflation factor to the ensemble analyses, and updated satellite bias correction coefficients (if requested) are output in the run directory. Below is an example of the files generated in the run directory from one of the EnKF regional test cases: 
\begin{scriptsize}    
\begin{verbatim}
analysis.ensmean            diag_amsua_n18_ges.mem003  diag_conv_ges.mem008           firstguess.mem002
analysis.mem001             diag_amsua_n18_ges.mem004  diag_conv_ges.mem009           firstguess.mem003
analysis.mem002             diag_amsua_n18_ges.mem005  diag_conv_ges.mem010           firstguess.mem004
analysis.mem003             diag_amsua_n18_ges.mem006  diag_conv_ges.mem011           firstguess.mem005
analysis.mem004             diag_amsua_n18_ges.mem007  diag_conv_ges.mem012           firstguess.mem006
analysis.mem005             diag_amsua_n18_ges.mem008  diag_conv_ges.mem013           firstguess.mem007
analysis.mem006             diag_amsua_n18_ges.mem009  diag_conv_ges.mem014           firstguess.mem008
analysis.mem007             diag_amsua_n18_ges.mem010  diag_conv_ges.mem015           firstguess.mem009
analysis.mem008             diag_amsua_n18_ges.mem011  diag_conv_ges.mem016           firstguess.mem010
analysis.mem009             diag_amsua_n18_ges.mem012  diag_conv_ges.mem017           firstguess.mem011
analysis.mem010             diag_amsua_n18_ges.mem013  diag_conv_ges.mem018           firstguess.mem012
analysis.mem011             diag_amsua_n18_ges.mem014  diag_conv_ges.mem019           firstguess.mem013
analysis.mem012             diag_amsua_n18_ges.mem015  diag_conv_ges.mem020           firstguess.mem014
analysis.mem013             diag_amsua_n18_ges.mem016  diag_gome_metop-a_ges.ensmean  firstguess.mem015
analysis.mem014             diag_amsua_n18_ges.mem017  diag_gome_metop-b_ges.ensmean  firstguess.mem016
analysis.mem015             diag_amsua_n18_ges.mem018  diag_omi_aura_ges.ensmean      firstguess.mem017
analysis.mem016             diag_amsua_n18_ges.mem019  diag_sbuv2_n16_ges.ensmean     firstguess.mem018
analysis.mem017             diag_amsua_n18_ges.mem020  diag_sbuv2_n17_ges.ensmean     firstguess.mem019
analysis.mem018             diag_conv_ges.ensmean      diag_sbuv2_n18_ges.ensmean     firstguess.mem020
analysis.mem019             diag_conv_ges.mem001       diag_sbuv2_n19_ges.ensmean     ozinfo
analysis.mem020             diag_conv_ges.mem002       diff.nc                        satbias_ang.out
anavinfo                    diag_conv_ges.mem003       enkf.log                       satbias_in
convinfo                    diag_conv_ges.mem004       enkf.nml                       satbias_out.int
diag_amsua_n18_ges.ensmean  diag_conv_ges.mem005       enkf.x                         satbias_pc
diag_amsua_n18_ges.mem001   diag_conv_ges.mem006       firstguess.ensmean             satinfo
diag_amsua_n18_ges.mem002   diag_conv_ges.mem007       firstguess.mem001              stdout
\end{verbatim}
\end{scriptsize}

This case was a regional analysis with WRF/ARW NetCDF backgrounds. In this case, 20 ensemble members are used to estimate ensemble covariance, and both conventional observations (prepbufr) and radiance observations (AMSU-A) are assimilated.

A brief introduction of the files is given in the table below:

\begin{table}[htbp]
\centering
\begin{tabular}{| l | l |}
\hline
stdout  &  A text output file. This is the most commonly used file to \\
             & check the analysis processes as well as basic and important \\
             &  information about the analyses. The contents of stdout are \\
             & explained in detail in Chapter 4 and users are encouraged to \\
             & read this file to become familiar with the order of EnKF  \\
             &   analysis processing.  \\
             \hline
firstguess. mem001-0??  &  ensemble members of WRF background, in NetCDF format. This \\
              &  is in the same format as the WRF forecast. The ensemble \\ 
               &  background of the analysis variables are extracted from the files.  \\
               \hline
firstguess.ensmean  &  ensemble mean of WRF background, in NetCDF format.  \\
\hline
 analysis. mem001-0??:  &  ensemble analysis if EnKF completes successfully.  \\
                                &  The format is the same as the background file.  \\   
                                \hline
  analysis.ensmean &  ensemble mean of analyses, in NetCDF format. \\
  \hline
   diag\_conv\_ges. mem001-0??:  &  diag files in binary for conventional and GPS RO \\
             &    observations, which contain the observations and their priors.       \\
             \hline 
  covinflate.dat  & Three-dimensional multiplicative inflation factor fields \\
  &  (from function inflate\_ens in module inflation). These  \\
    &  inflation fields can be visualized using plotting software.  \\      
    \hline
    *info (convinfo, satinfo,\ldots):  &  info files that control data usage. Please \\
    &  see GSI User guide Chapter 4 for details.\\
         \hline
      satbias\_in  & the input coefficients of bias correction \\
      & for satellite radiance observations.\\       
      \hline
        satbias\_pc    &  the input coefficients of satellite radiance bias \\
        &  correction for passive channels.\\        
        \hline                 
\end{tabular}
\end{table}

    
    
